Skip to content

fix(redshift): terminate running queries when SQLMesh is interrupted - #6144

Draft
gandeevanraghuramandd wants to merge 4 commits into
SQLMesh:mainfrom
gandeevanraghuramandd:graghura/redshift-terminate-queries-on-interrupt
Draft

gandeevanraghuramandd wants to merge 4 commits into
SQLMesh:mainfrom
gandeevanraghuramandd:graghura/redshift-terminate-queries-on-interrupt

Conversation

@gandeevanraghuramandd

@gandeevanraghuramandd gandeevanraghuramandd commented Oct 14, 2026 •

Copy link
Copy Markdown

Description

On Ctrl-C during a concurrent run, SQLMesh exits without stopping the queries its worker threads started. Redshift keeps running them after the client disconnects, holding their locks until they finish. The workers also keep going after teardown and can commit part of a model on a new connection.

This PR:

  • stops waiting for worker threads when the run is interrupted
  • terminates the Redshift sessions of running queries (pg_terminate_backend) in recycle() and close()
  • fails a worker's transaction if another thread closed its connection mid-transaction, instead of letting it continue on a new connection
  • stops abandoned workers before they start a snapshot's transaction

Test Plan

Added unit tests for each change. Not yet tested against a real Redshift cluster.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Made with Cursor

On Ctrl-C, SnapshotEvaluator.concurrent_context() recycles the adapter
before the context closes it. ThreadLocalConnectionPool.close_all() clears
every thread's attributes, so close() no longer found the backend PIDs of
queries still running on worker threads and left them running on Redshift.

RedshiftEngineAdapter now terminates running queries in both recycle() and
close(), before the pool is closed. If the calling thread was interrupted
mid-query or mid-transaction, its connection is closed and the sessions are
terminated from a new one.

Also removes the SIGTERM-to-SIGINT mapping in the CLI, which is deferred to
a follow-up so this change is scoped to Ctrl-C.
When the thread-local connection pools close another thread's connection
while it's in the middle of a transaction (e.g. when Ctrl-C recycles the
pool while workers are still running), they used to forget the
transaction. The worker's next statement then ran on a new connection in
a new transaction and could be committed without the statements that
were rolled back with the closed connection.

The transaction is now marked as broken instead. The worker's next
statement fails without opening a new connection, its rollback doesn't
touch the database, and committing it fails. Rollback also ends the
transaction now even if it fails.
When SQLMesh is interrupted (e.g. by Ctrl-C), worker threads aren't
waited for. A worker that hadn't started evaluating or creating a
snapshot yet would carry on after teardown on a new connection and run
the snapshot's statements anyway.

concurrent_context() now sets an event when it's interrupted, and
workers check it before a snapshot's non-transactional pre-statements
and before its main transaction. Work after the main transaction
commits isn't stopped, so the interval of committed data is still
recorded.
@gandeevanraghuramandd gandeevanraghuramandd changed the title Fix: terminate running Redshift queries when SQLMesh is interrupted fix(redshift): terminate running queries when SQLMesh is interrupted Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant